A comparison of common metrics used to quantify the effectiveness of conservation interventions

Introduction

Coping with globally widespread human pressures on the environment demands the urgent promotion of evidence-based conservation, which is the use of best available evidence to foster informed decision-making in practical conservation (Pullin et al., 2004; Sutherland et al., 2004; Sutherland & Wordley, 2017; Altringham, Berthinussen & Wordley, 2020). This concept ensues from, and follows the principles of, the effectiveness revolution which began in the 1970s in medicine (Pullin & Knight, 2001). In other words, evidence-based conservation strives to collect, analyze, interpret, synthesize and distribute information on the effectiveness of interventions applied to mitigate threats and improve the status of (semi-)natural ecosystems and their components (Westgate et al., 2018; Altringham, Berthinussen & Wordley, 2020). Evidence is subject to rigorous scientific assessments and it should be widely published to stimulate replications and reduce the research-implementation gap (Toomey, Knight & Barlow, 2017; Burivalova et al., 2019). Comparisons of several interventions in the same setting are very useful to select the most effective interventions for further applications (Smith et al., 2014; Schmidt et al., 2019). Active development of open-access web resources (e.g., Conservation Evidence—www.conservationevidence.com and the Collaboration for Environmental Evidence—www.environmentalevidence.org), professional meetings, analytical tools and outreach awareness-raising is aimed at the wider use of evidence in conservation actions (Sutherland & Wordley, 2017; Westgate et al., 2018; Sutherland et al., 2019; Pullin et al., 2020).

Deeper understanding of the importance of intervention effectiveness leads to more scrutiny over the issues of study design and statistical analysis. It is generally agreed that experiments must be controlled and case-control studies, such as comparisons before and after the intervention and comparisons with and without the intervention, are less reliable because they do not have properly matched intervention-free controls to compare with (Burivalova et al., 2019; Christie et al., 2019). Arguably, the most reliable study designs are before-after-control-impact (BACI), control trials and cross-over, especially when study objects are randomized and experimental procedures are blinded to minimize biases (Adams, Barnes & Pressey, 2019; Christie et al., 2019; Treves et al., 2019). A number of metrics have been used to quantify the effectiveness of interventions, including the relative risk (Eklund et al., 2017; Khorozyan & Waltert, 2019a; Khorozyan & Waltert, 2019b; Bruns, Waltert & Khorozyan, 2020; Khorozyan et al., 2020), odds ratio (Knarrum et al., 2006; Woodroffe et al., 2007; Wielgus & Peebles, 2014), magnitude of change (Jones & Schmitz, 2009; Green et al., 2013; Miller et al., 2016) and Hedges’ d (Van Eeden et al., 2017).

Despite progress in developing the methodology of evidence-based conservation, very little understanding has been obtained regarding differences in effectiveness estimates produced by different metrics. Scientists usually use a single metric per study and it remains unclear whether their estimates are sufficiently accurate or produce biases (underestimation and overestimation) for a variety of reasons. The relative risk and the odds ratio are very similar when events are rare (Stare & Maucort-Boulch, 2016), for example, when predators kill livestock and the effectiveness of livestock protection interventions is evaluated (Khorozyan & Waltert, 2019a; Bruns, Waltert & Khorozyan, 2020). Van Eeden et al. (2018) compiled three independent meta-analyses, each of which used a different metric (relative risk, magnitude of change and Hedges’ d), to assemble current information on the effectiveness of livestock protection interventions against predators, but did not compare these metrics from the same case studies.

It is of particular practical importance and scientific interest to compare the metrics calculated from the same standardized matrix produced from the same study. Among the most common matrices is the 2 × 2 contingency table which summarizes the distribution of dichotomous outcomes, such as species A present and species A absent, across the treatment (with intervention) and control (without intervention) samples (Stare & Maucort-Boulch, 2016). The contingency table approach can be used to calculate the relative risk, odds ratio and the magnitude of change with no need for additional information. The effects of treatment and control samples and their relationships on metrics are key to identify possibilities for accurate and unbiased estimation of effectiveness. Larger samples improve the precision of effectiveness estimates (Smith et al., 2014; Christie et al., 2019), but this is not sufficient and the relationships between treatment and control samples may uncover new effects worthy of further investigations. As a researcher is responsible for setting treatment and control sample sizes before the study, this knowledge may provide useful rules-of-thumb for study design, implementation and data analysis.

In this study, I used a simulated dataset to calculate the relative risk, odds ratio and magnitude of change from the 2 × 2 contingency table, estimate their disparity, and find sample characteristics significantly affecting this disparity. I did not consider Hedges’ d as this metric requires additional quantitative information (mean and standard deviation) from treatment and control samples, which is not available from contingency tables (Nakagawa & Cuthill, 2007). Four case studies from different conservation interventions are provided and described in order to illustrate and support the results.

Materials and Methods

Framework

The 2 × 2 contingency table of the numbers of target and alternative outcomes in treatment and control samples was a basis of this study (Fig. 1). The target outcomes are the outcomes that interventions strive to change, otherwise they are designated as the alternative outcomes. As interventions aim to reduce negative target outcomes or add positive target outcomes, they were provisionally named as reduction-aimed interventions and addition-aimed interventions. An example of reduction-aimed intervention is livestock protection that aims to reduce or eliminate livestock losses to predators (Eklund et al., 2017) and an example of addition-aimed intervention is the establishment of protected areas to increase species presence (Da Silva et al., 2018). In these two examples, the target outcomes are the number of lost livestock and the number of sampling units (e.g., grids or transects) with species presence. The alternative outcomes are the number of surviving livestock and the number of sampling units with species absence. The treatment samples are those affected by interventions and the control samples, also known as counterfactuals, are not affected.

The purpose of data analysis from the contingency table is to determine the effectiveness of interventions, that is, whether they significantly decrease the target outcomes for reduction-aimed interventions or significantly increase the target outcomes for addition-aimed interventions. In Fig. 1, N_t1 stands for the number of target outcomes in treatment sample, N_t2 for the number of alternative outcomes in treatment sample, N_c1 for the number of target outcomes in control sample and N_c2 for the number of alternative outcomes in control sample. The treatment sample size is N_t = N_t1 + N_t2 and the control sample size is N_c = N_c1 + N_c2.

Results

Disparity distances between OR% and the other two metrics were significantly affected only by the odds of target outcomes in treatment sample (N_t1/N_t2), which is natural as OR% is calculated from these odds (see Eq. (2) above). Therefore, only RR% and D% were considered further in this study. The only outcome ratio significantly affecting the disparity distance between RR% and D% was the ratio of the number of target outcomes in treatment sample (N_t1) to that in control sample (N_c1), that is, N_t1/N_c1 (Fig. 2). This effect was significant (F_1,480 = 1,194.49, p < 0.001), precise and positive (ß₁ = 51.402 ± 1.487, 95% CI [48.480–54.324]) and strong (r² = 0.713), but not accurate (nRMSE = 575.63%). As D% is calculated from N_t1/N_c1, its line was statistically perfect (r² = 1). The effect of N_t1/N_c1 on RR% was also significant (F_1,489 = 2,081.50), precise and positive (ß₁ = 52.571 ± 1.152, 95% CI [50.307–54.835]), strong (r² = 0.810) and accurate (nRMSE = 3.33%) (Fig. 2). The slopes of D% and RR%, which embraced the disparity distance within, were statistically different (ANCOVA: F_1,982 = 2,036.80, p < 0.001).

When N_t1/N_c1 was less than 1, there was a significantly higher number of cases with D% < RR% (n = 179) than with D% > RR% (n = 75), and this difference was significant (χ² = 42.583, p < 0.001). The estimate D% was significantly lower than RR% (Mann–Whitney U = 25,913.5, p < 0.001). These inequalities depended on the ratio of treatment sample size (N_t) to control sample size (N_c): |D%| < |RR%| when N_t/N_c was higher than 1 and |D%| > |RR%| when N_t/N_c was less than 1. When N_t/N_c is equal to 1, D% = RR%. The absolute estimates are provided because D% and RR% can be both positive and negative. The general scheme of underestimations, overestimations and equalities of D% and RR% is shown in Table 1.

When N_t1/N_c1 was higher than 1, there was a significantly higher number of cases with D% > RR% (n = 172) than with D% < RR% (n = 64), and this difference was significant (χ² = 49.424, p < 0.001). The estimate D% was significantly higher than RR% (Mann–Whitney U = 18,779.0, p < 0.001). Like in the previous scenario, these relationships depended on N_t/N_c: |D%| > |RR%| when N_t/N_c was higher than 1 and |D%| < |RR%| when N_t/N_c was less than 1. Again, when N_t/N_c is equal to 1, D% = RR% (Table 1).

When N_t1/N_c1 is equal to 1, D% indicates no effect of interventions regardless of relationships between N_t and N_c whereas RR% indicates a decrease or increase of target outcomes depending on sample sizes. When N_t = N_c, both D% and RR% estimate no effect of interventions (Table 1).

Discussion

This study has shown that the relative risk and the magnitude of change produce identical estimates of intervention effectiveness only when treatment and control samples are equal, or when the number of target outcomes in treatment sample is zero (Table 1). In other situations, the magnitude of change gives overestimates or underestimates, depending on relationships between treatment and control sample sizes. These patterns were demonstrated by means of theoretical simulations and practical case studies, for both reduction-aimed and addition-aimed conservation interventions. Overestimation takes place when the relationships between treatment and control sample sizes, and between the numbers of target outcomes in treatment and control samples, are in the same direction—that is, both are lower, or both are higher, in treatment than in control samples (Table 1). When these relationships go in opposite directions, the magnitude of change gives underestimation of effectiveness. The higher the dissimilarity between treatment and control sample sizes, the higher the magnitude of overestimation or underestimation, which is best illustrated by case studies 1 and 4 (Knarrum et al., 2006; Andruk, 2014). Disparity between the two metrics further increases with the increasing ratio of the number of target outcomes in the treatment sample to that in the control sample (Fig. 2).

Information about the accuracy of different effectiveness metrics under variable relationships between treatment and control sample sizes is practically important because the determination of sample size is the researcher’s prime task to design a good study (Smith et al., 2014; Christie et al., 2019). When samples are subject to study design, as in the cases with bell-collared and uncollared ewes (case study 1; Knarrum et al., 2006), net-covered and uncovered turtle nests (case study 2; Sampaio, 2018) and swift nest boxes with and without nest forms (case study 3; Newell, 2019), they should be set to be equal or similar between treatment and control. In contrast, when sample size is beyond control, for example, it represents the number of sprouts in case study 4 (Andruk, 2014), it is not possible to secure the equality of treatment and control sample sizes. Some difference between treatment and control sample sizes is still acceptable, as in case study 3, but its increase leads to inaccurate estimation of intervention effectiveness by the magnitude of change. As a result, this study not only supports an appeal for balanced study designs in ecological research (Fletcher & Underwood, 2002), but also indicates the conditions when biases in estimation are possible.

The only situation when the relationships between treatment and control samples do not matter is when the number of target outcomes in treatment sample is 0. This is the best outcome for reduction-aimed interventions which is quantified by the relative risk and the magnitude of change as the maximum reduction of target outcome by 100% (Table 1). Interventions applied to reduce, inter alia, the rates of livestock depredation (Eklund et al., 2017; Van Eeden et al., 2018), deforestation (Burivalova et al., 2019) and poaching (Kragt et al., 2020) are all aimed at reduction of target outcomes and their ultimate goal is to nullify them. However, such maximum effectiveness cannot be assumed a priori even for interventions which often show excellent results, like electric fences for reduction of livestock losses (Khorozyan & Waltert, 2019a, 2019b), because the effectiveness is site-specific. Therefore, proper setting of treatment and control sample sizes is essential anyway. These sample sizes, as well as the numbers of target and alternative outcomes as their sub-samples (Fig. 1), should be explicitly reported by researchers to allow independent evaluation of intervention effectiveness.

If designed treatment and control samples are unlikely to be similar, it is most reasonable to expect the treatment sample to be smaller than the control sample due to significant efforts and resources invested to the process of treatment (Eklund et al., 2017; Snijders et al., 2019). This was confirmed by the first three case studies where sample sizes were determined by researchers (Knarrum et al., 2006; Sampaio, 2018; Newell, 2019). In this situation, three scenarios are possible: (1) if the number of target outcomes in treatment sample is lower than that in control sample—overestimation by the magnitude of change; (2) if the number of target outcomes in treatment sample is higher than that in control sample—underestimation by the magnitude of change; and (3) if the numbers of target outcomes in treatment and control samples are equal—the magnitude of change estimates no effectiveness and the relative risk indicates an increase in target outcome (Table 1).

Why are overestimation and underestimation specified for the magnitude of change in regard to the relative risk, but not vice versa? The point is that the magnitude of change does not consider sample sizes and is thus a less reliable metric. The relative risk is a ratio of the probability of target outcome with the intervention to that probability without the intervention (Eklund et al., 2017; Khorozyan & Waltert, 2019a), so it represents a balanced metric accounting for treatment and control sample sizes. It is one of the main effectiveness metrics in medicine (Stare & Maucort-Boulch, 2016), a discipline which is well advanced in research of evidence-based intervention effectiveness (Pullin & Knight, 2001; Sutherland et al., 2004; Sutherland & Wordley, 2017; Altringham, Berthinussen & Wordley, 2020). In contrast, the magnitude of change is based only on the difference between the numbers of target outcomes (Jones & Schmitz, 2009; Green et al., 2013; Miller et al., 2016), yet sample sizes are critically important and their ignorance can drastically change the effectiveness estimates. A vivid example is the situation when the number of target outcomes in treatment sample is equal to that in control sample. In this case, the magnitude of change fails to detect the effectiveness and quantifies it as 0, whereas the relative risk estimates the effectiveness to increase or decrease depending on sample sizes (Table 1). In extreme cases of imbalanced samples, with high dissimilarity between treatment and control samples, the magnitude of change can even show an opposite direction of effectiveness in comparison with the relative risk. For example, imagine a fictitious case when the numbers in Table 2 are changed as follows: number of killed ewes with bells—1, number of surviving ewes with bells—7, number of killed ewes without bells—3 and number of surviving ewes without bells—97. So, the treatment sample size is 8 and the control sample size is 100. In this case, the magnitude of change shows a reduction of depredation rate by 66.7%, but the relative risk shows an increase by 316.7%.

I hope that this study will stimulate more active implementation of well-designed scientific and practical research in evidence-based conservation so that to make tangible and long-lasting positive changes in the state of the environment.

Conclusions

Researchers should set equal treatment and control sample sizes in order to produce identical estimates of intervention effectiveness by the relative risk and the magnitude of change. When treatment and control samples are unequal, these estimates are biased if produced by the magnitude of change and the relative risk should be used instead. As setting equal treatment and control samples can be problematic for practical reasons, I encourage researchers and practitioners to use the relative risk in estimation of intervention effectiveness. This will not take additional efforts as both metrics are calculated from the same contingency table. Researchers should report treatment and control sample sizes, and the numbers of target and alternative outcomes within these samples, to let the readers evaluate intervention effectiveness independently. Overall, this approach can strengthen study design and data analysis by scientists, increase the reliability of results for practitioners, and facilitate decision-making by conservation managers based on accurate information.

Supplemental Information